Dynamic user interface adaptable to multiple input tools

ABSTRACT

A computing device having a touch-sensitive surface and a display, detects a stylus input on the touch-sensitive surface while displaying a user interface. A first operation is performed in the user interface in accordance with a determination that the stylus input includes movement of the stylus across the touch-sensitive surface while the stylus is detected on the touch-sensitive surface. A second operation different from the first operation is performed in the user interface in accordance with a determination that the stylus input includes rotation of the stylus around an axis of the stylus while the stylus is detected on the touch-sensitive surface. A third operation is performed in the user interface in accordance with a determination that the stylus input includes movement of the stylus across the touch-sensitive surface and rotation of the stylus around an axis of the stylus while the stylus is detected on the touch-sensitive surface.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/445,108, filed Jun. 18, 2019, which is a continuation of U.S.application Ser. No. 15/607,186, filed May 26, 2017, now U.S. Pat. No.10,342,549, which is a continuation of U.S. application Ser. No.14/616,532, filed Feb. 6, 2015, now U.S. Pat. No. 9,665,206, which is acontinuation of U.S. application Ser. No. 14/030,682, filed Sep. 18,2013, abandoned, each of which is hereby incorporated by reference inits entirety.

BACKGROUND

The present disclosure relates generally to a user interface and inparticular to a user interface element that dynamically adapts todifferent user input tools.

Recent years have seen a proliferation of touchscreen-based computingdevices, including mobile phones, tablets, and hybrid computers operableas a tablet or a laptop. These devices generally incorporate a displayarea overlaid with a touch-sensitive overlay (e.g., a resistive orcapacitive overlay). The display can present a user interfaceincorporating various input and/or output elements, such as virtualbuttons and video, still-image, or text presentations. The overlay candetect contact by an object such as a finger or stylus or the like andcan determine the portion of the display that was contacted. Based onthe contacted portion of the display and the currently presentedinterface, the device can infer a user request or instruction.

SUMMARY

A graphical user interface element displayed by a computing device canbe dynamically adapted to different user input tools, for example afingertip or a stylus.

The following detailed description together with the accompanyingdrawings will provide a better understanding of the nature andadvantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a computing device and a stylus according to an embodimentof the present invention.

FIG. 2 is a simplified block diagram of a system including a computingdevice and a stylus according to an embodiment of the present invention.

FIG. 3 illustrates a user interface screen that can be presented by acomputing device according to an embodiment of the present invention.

FIG. 4 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 5 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 6 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 7 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 8 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 9 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 10 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 11 illustrates another user interface screen that can be presentedby a computing device according to an embodiment of the presentinvention.

FIG. 12 is a flow diagram of a process for rendering a user interfaceaccording to an embodiment of the present invention.

FIG. 13 is a flow diagram of a process for determining stylus stateaccording to an embodiment of the present invention

DETAILED DESCRIPTION

Certain embodiments of the present invention relate to computingdevices, such as mobile computing devices, that have touchscreen userinterfaces operable by different input tools, such as a user's finger ora stylus held by the user. These different input tools can contact thetouchscreen with different-sized contact areas. For example, a styluscan have a fine point comparable to a pen or pencil, while a fingertipis comparatively blunt. Consequently, the stylus can contact a smallerarea of the touchscreen than a finger. Further, a user may finddifferent gestures more or less comfortable depending on the input tool.For example, while twisting or rotating a finger while keeping thefingertip in contact with the screen may be uncomfortable, twisting orrotating a stylus may feel more natural and comfortable. For reasonssuch as these, a user input interface that is optimized for fingertipcontact may not be optimal for other input tools such as a stylus, andvice versa.

Accordingly, certain embodiments provide graphical user interfacesand/or graphical user interface elements that can be dynamically andautomatically altered depending on the type of input tool in use. Thetype of input tool can be automatically detected, and the detectionprocess can be transparent to the user. For example, a stylus can have atouch sensor built into its body, covering an area where a user's handis expected to contact the stylus while in use. In response to detectingcontact with the touch sensor, the stylus can communicate to thecomputing device that the stylus is in an “active” state, and thecomputing device can present a graphical user interface with inputelements adapted for stylus input. When the touch sensor of the stylusceases to detect contact, the stylus can communicate to the computingdevice that the stylus is in an “idle” state, and the computing devicecan present a graphical user interface with input elements adapted forfingertip input.

Examples of adaptations for a particular input tool can include changingthe number of input elements displayed on the screen (e.g., fewer andlarger input elements for fingertip input than for stylus input),changing the arrangement of input elements (e.g., elements placed closertogether for stylus input), changing a control-adjustment paradigm(e.g., slider buttons for fingertip input and rotation-based controlssuch as a virtual set-screw for stylus input), changing the manner ofinput (e.g., stroke-based input for stylus versus virtual keyboard forfingertip input), and so on. Additional examples are described below.

FIG. 1 shows a mobile computing device 100 (a tablet computer in thisexample) and a stylus 102 according to an embodiment of the presentinvention. Computing device 100 can include a touchscreen display area104 with a surrounding bezel 106. A control button 108 can be provided,e.g., to return touchscreen display area 104 to a “home” screen. Stylus102 can include a barrel portion 110 and a tip portion 112. Varioussensors can be disposed on or within stylus 102. Shown by way of exampleare touch sensors 114 disposed along barrel portion 110. In someembodiments, touch sensors 114 can be capacitive sensors that respond tocontact with human skin by varying a capacitance; electronic circuitsinside stylus 102 can sense the change in capacitance and thereby detectcontact. Other types of touch sensors can also be used, such as pressuresensors that can detect pressure of the user's grip on the stylus,proximity sensors that can detect patterns of proximity consistent withbeing in a user's hand, and so on; one or more types of touch sensorscan be used in any combination. Touch sensors 114 can be distributedsuch that it is highly likely that whenever a user picks up stylus 102with intent to use it, at least one sensor 114 will register thecontact. Stylus 102 can but need not be capable of distinguishingcontact with different touch sensors 114.

Other sensors disposed on stylus 102 can include contact sensors 116,118 disposed at the tip end 120 and/or the reverse end 122. Sensors 116and 118 can be configured to detect proximity to and/or contact withcomputing device 100. In some embodiments, tip end 120 can be tapered asshown and can come to a point or nearly to a point (with a small crosssectional area relative to a finger), while reverse end 122 can be bluntor slightly rounded, making the two ends visually distinguishable. Theparticular size and shape of tip end 120 and/or reverse end 122 can bemodified as desired. In embodiments described herein, it is assumed thattip end 120 provides a contact area with touchscreen display 104 that ismeasurably smaller than an average human finger; however, this is notrequired. Sensors 116, 118 can be any sensors usable to detect orconfirm contact with touchscreen display 104. In some embodiments,sensors 116 and 118 can modify a surface capacitance of tip end 120 orrear end 122 on contact; if touchscreen display 104 is a capacitivetouchscreen, this change can be sensed directly by computing device 100.

Stylus 102 can be an active electronic device capable of communicatingsignals from its sensors (e.g., sensors 114, 116, 118) to computingdevice 100. Wired or wireless communication channels can be used;specific examples are described below with reference to FIG. 2.Computing device 100 can use the sensor information transmitted fromstylus 102 together with its own state information to adapt its behaviorbased on use of stylus 102. For example, stylus 102 can detect when itis picked up by a user, e.g., using touch sensors 114. When stylus 102is picked up, it can transmit a signal to computing device 100indicating that stylus 102 is now in an “active” state. (As used herein,an “active” state of the stylus refers to a state in which the sensorsof the stylus indicate that it is being held by a user, while an “idle”state of the stylus refers to a state in which the sensors of the stylusdo not indicate that it is being held by a user.) In response to thissignal, computing device 100 can adapt its interface accordingly, forinstance by modifying the size, spacing, and/or number of visible inputelements to better make use of the relatively small contact area of tipportion 120 of stylus 102.

It will be appreciated that the computing device and stylus of FIG. 1are illustrative and that variations and modifications are possible. Forexample, while a tablet computer is shown, computing devices can vary insize, form factor, and capabilities; other types of computing devicescan include mobile phones, laptop computers, hybrid computers thatincorporate a combination of touchscreen and keyboard interfaces, and soon. Computing devices can be but need not be portable. Computing devicescan also vary as to capabilities; for example, various computing devicesmay or may not provide mobile voice and/or data services (e.g., via acellular network), Internet connectivity (e.g., via Wi-Fi or otherwireless or wired interfaces or via a cellular data network), GlobalPositioning System services, media player capability, personal datamanagement services (e.g., calendar, contacts, notes, reminders), and/orthe ability to run various application programs including third-partyapplication programs to provide a wide range of user interactions (e.g.,interactive drawing, word processing, spreadsheets, calculators,information storage and retrieval, games, activity monitoring, documentdisplay and reading, etc.).

A stylus can have any size and form factor. In some embodiments, astylus can be suitably sized and shaped to be held comfortably in ahuman hand; however, it is not required that the stylus be handheld oreven that a user of the stylus have hands. As long the stylus hassensors able to detect when it is or is not being held or actively usedby a user, active and idle states as described herein can bedistinguished. For example, in some embodiments, a stylus can includeinternal accelerometers (or other motion sensors) in addition to orinstead of touch sensors; whether the stylus is being held by a user canbe determined by comparing accelerometer data to patterns of motionsexpected when a stylus is being held by a person (including, e.g., smallvibration due to the natural unsteadiness of human hands).

FIG. 2 is a simplified block diagram of a system 200 including computingdevice 202 and stylus 204 according to an embodiment of the presentinvention. In this embodiment, computing device 202 (e.g., implementingcomputing device 100 of FIG. 1) can provide computing, communicationand/or media playback capability. Computing device 200 can includeprocessing subsystem 210, storage device 212, touchscreen interface 214,network interface 216, stylus interface 218, and accessory input/output(I/O) interface 220. Computing device 202 can also include othercomponents (not explicitly shown) such as a battery, power controllers,and other components operable to provide various enhanced capabilities.

Storage device 212 can be implemented, e.g., using disk, flash memory,or any other non-transitory storage medium, or a combination of media,and can include volatile and/or non-volatile media. In some embodiments,storage device 212 can store data objects such as audio files, videofiles, image or artwork files, information about a user's contacts(names, addresses, phone numbers, etc.), information about a user'sscheduled appointments and events, notes, and/or other types ofinformation or data files. In some embodiments, storage device 212 canalso store one or more application programs to be executed by processingsubsystem 210 (e.g., video game programs, personal informationmanagement programs, media playback programs, etc.) and/or one or moreoperating system programs or other firmware to implement and supportvarious device-level capabilities including generation of graphical userinterfaces and processing of user input.

Touchscreen interface 214 can be implemented, e.g., as an LCD displaywith a touch-sensitive overlay such as a resistive or capacitive overlayand can be operable using various input tools such as fingertip, stylus,and so on. In some embodiments, computing device 202 can also includeother user input devices such as a touch pad, scroll wheel, click wheel,dial, button, switch, keyboard, keypad, microphone, or the like, and/oroutput devices such as indicator lights, speakers, headphone jacks, orthe like, together with supporting electronics (e.g., digital-to-analogor analog-to-digital converters, signal processors, or the like). A usercan operate touchscreen 214 (and any other input devices that may bepresent) to interact with computing device 202.

Processing subsystem 210 can be implemented as one or more integratedcircuits, e.g., one or more single-core or multi-core microprocessors ormicrocontrollers, examples of which are known in the art. In operation,processing subsystem 210 can control the operation of computing device202. In various embodiments, processing subsystem 210 can execute avariety of programs in response to program code and can maintainmultiple concurrently executing programs or processes. At any giventime, some or all of the program code to be executed can be resident inprocessing subsystem 210 and/or in storage media such as storage device212.

Network interface 216 can provide voice and/or data communicationcapability for computing device 202. In some embodiments networkinterface 216 can include radio frequency (RF) transceiver componentsfor accessing wireless voice and/or data networks (e.g., using cellulartelephone technology, advanced data network technology such as 3G orEDGE, Wi-Fi (IEEE 802.11 family standards), or other mobilecommunication technologies, or any combination thereof), components forshort-range wireless networking (e.g., using Bluetooth standards), GPSreceiver components, and/or other components. In some embodimentsnetwork interface 216 can provide wired network connectivity (e.g.,Ethernet) in addition to or instead of a wireless interface. Networkinterface 216 can be implemented using a combination of hardware (e.g.,driver circuits, antennas, modulators/demodulators, encoders/decoders,and other analog and/or digital signal processing circuits) and softwarecomponents.

Stylus interface 218 can receive communications from stylus 204 and/orsend communications to stylus 204. For example, stylus interface 218 canuse a wireless communication protocol such as Bluetooth or Bluetooth LEthat provides for serial data transfer. A serial protocol with auniverse of messages and message formats can be defined. For instance,depending on implementation, stylus 204 can send any or all of thefollowing messages: a “stylus active” message indicating that stylus 204is in the active state; a “stylus idle” message indicating that stylus204 is in the idle state; and/or a message containing sensor data fromone or more sensors of stylus 204. Other messages or combinations ofmessages can also be supported.

Accessory I/O interface 220 can allow computing device 202 tocommunicate with various accessories. For example, accessory I/Ointerface 220 can support connections to a computer, an externalkeyboard, a speaker dock or media playback station, a digital camera, aradio tuner, an in-vehicle entertainment system or head unit, anexternal video device, a memory card reader, and so on. In someembodiments, accessory I/O interface 220 can support wirelesscommunication (e.g., via Wi-Fi, Bluetooth, or other wirelesstransports). The same wireless transceiver hardware as network interface216 and/or stylus interface 218 can be used for both networking andaccessory communication. Additionally, in some embodiments, accessoryI/O interface 220 can include a connector as well as supportingcircuitry. In some instances, stylus 204 can connect via accessory I/Ointerface 220, and a separate stylus interface is not required.

Through suitable programming, processing subsystem 210 can providevarious functionality for computing device 202. For example, processingsubsystem 210 can execute operating system code and/or application codeto render a graphical user interface (“GUI”) image that includes one ormore user-operable input elements (e.g., virtual buttons, knobs,sliders, set-screws, data-entry boxes, etc.). Processing subsystem 210can also receive and interpret messages from stylus 204 via stylusinterface 208 and can use the messages to modify the rendering of theGUI image based on the received messages. For instance, as describedbelow, the number, size, arrangement, and/or control type of inputelements can be modified. Processing subsystem 210 can also executeother programs to control other functions of computing device 202,including application programs that may be stored in storage device 212.

Stylus 204 can include various sensors such as contact sensors 232,touch sensors 234, and accelerometer(s) 236; a signal processor 238; anda transmitter (or transceiver) 240. Stylus 204 can also include othercomponents (not explicitly shown) such as a battery, power controllers,and other components operable to provide various enhanced capabilities.

Contact sensor(s) 232 (e.g., implementing sensors 116, 118 of FIG. 1)can be configured to detect contact of specific areas of stylus 204 witha touchscreen display of computing device 202 and to generatecorresponding electronic signals. For example, resistive or capacitivecontact sensors can be used. In some embodiments, contact sensors 232can generate a signal (e.g., a change in resistance or capacitance) thatthe touchscreen display of computing device 202 can detect as contact.

Touch sensor(s) 234 (e.g., implementing sensors 114 of FIG. 1) can beconfigured to detect touching of side surfaces of stylus 204 (e.g.,barrel portion as shown in FIG. 1.) and to generate correspondingelectronic signals. Any type of sensor can be used, including capacitivesensors, resistive sensors, pressure-based sensors or the like.

Accelerometer(s) 236 can be configured to detect motion of stylus 204and generate corresponding electronic signals. A three-axis MEMS-basedaccelerometer or the like can be used.

Transmitter 240 can be configured to send messages from stylus 204 tostylus interface 218 of computing device 202. In the example shown,transmitter 240 and stylus interface 218 support wireless communication,and transmitter 240 can include a combination of hardware (e.g., drivercircuits, antennas, modulators/demodulators, encoders/decoders, andother analog and/or digital signal processing circuits) and softwarecomponents. In addition or instead, transmitter 240 can provide aphysical connection to computing device 202 and can include a connector,a cable, drivers, and other analog and/or digital signal processingcircuits. In some embodiments, transmitter 240 can provide one-waycommunication from stylus 204 to computing device 202; in otherembodiments, a two-way transceiver can be used, and stylus 204 can bothreceive messages from computing device 202 and send messages tocomputing device 202.

Signal processor 238 can detect electronic signals from the varioussensors including contact sensor(s) 232, touch sensor(s) 234, and/oraccelerometer 236 and can generate messages to be communicated tocomputing device 202 via transmitter 240. In various embodiments, themessages can include a representation of the electronic sensor signalsand/or other information derived from the electronic sensor signals suchas whether the stylus is in the active or idle state, whether (and whatportion of) the stylus is in contact with the touchscreen interface ofcomputing device 202, and so on. Status messages such as battery statusinformation, stylus identifying information, and the like, can also becommunicated using transmitter 240.

In some embodiments, stylus 204 can implement a wireless communicationprotocol (e.g., using transmitter 240) that provides proximity detectionbased on signal strength. For example, using Bluetooth LE, stylus 204and computing device 202 can establish a pairing, and stylus 204 (orcomputing device 202) can use the strength of Bluetooth LE signals fromcomputing device 202 (or stylus 204) to estimate proximity between thedevices. Signals from contact sensor(s) 232 in combination with theproximity estimate can be used to determine whether contact with thetouchscreen has occurred.

It will be appreciated that the system configurations and componentsdescribed herein are illustrative and that variations and modificationsare possible. The computing device and/or stylus may have othercapabilities not specifically described herein (e.g., mobile phone,global positioning system (GPS), broadband data communication, Internetconnectivity, etc.).

Further, while the computing device and stylus are described herein withreference to particular blocks, it is to be understood that these blocksare defined for convenience of description and are not intended to implya particular physical arrangement of component parts. Further, theblocks need not correspond to physically distinct components, and thesame physical components can be used to implement aspects of multipleblocks. Blocks can be configured to perform various operations, e.g., byprogramming a processor or providing appropriate control circuitry, andvarious blocks might or might not be reconfigurable depending on how theinitial configuration is obtained. Embodiments of the present inventioncan be realized in a variety of apparatus including electronic devicesimplemented using any combination of circuitry and software.

FIG. 3 illustrates one example of a user interface screen 300 that canbe presented by a computing device (e.g., computing device 100 ofFIG. 1) according to an embodiment of the present invention. In thisexample, user interface screen 300 supports creation and editing of apresentation that can incorporate text, images, and other media. Apresentation page is rendered in main section 302; in this example, thepresentation page includes an image 304 and accompanying text 306. Acontrol section 308 can provide various user operable input elementsthat the user can select (e.g., by touching) to modify the presentationpage. For example, when image 304 is highlighted, selecting controlbutton 310 can cause an image-formatting menu to be displayed; when text306 is highlighted, selecting control 308 can cause a text-formattingmenu to be displayed. In some embodiments, user interface screen 300 canbe presented when the stylus is in idle mode, and controls in section308 and/or menus that are displayed can be optimized for fingertipoperation.

FIG. 4 illustrates an example of a user interface screen 400 that can bepresented when a user, presented with user interface screen 300 of FIG.3, highlights image 304 (as shown by highlight outline 401) and selectsbutton 310. In response, computing device 100 can present an imageformatting menu 402 as an overlay on top of a portion of presentationarea 302. In this example, menu 402 provides various options for displayeffects that can be associated with image 304. For instance, the usercan select, using virtual switch 404, whether image 304 should cast avirtual shadow on the presentation page; if shadowing is turned on, thensection 406 can present selectable options for the type and orientationof the shadow. Similarly, the user can select, using virtual switch 408,whether a reflection should be provided; if reflection is turned on (notshown in this example), options for the reflection can be provided.Slider 410 can be moved laterally to set the desired opacity from 0%(fully transparent) to 100% (fully opaque).

It should be noted that menu 402 can be displayed when the stylus is inthe idle state, and the various control elements presented in menu 402can be adapted to be easily operated by a user's finger making gestureson a touchscreen, e.g., by making the controls of appropriate size fortypical fingertips to unambiguously touch one control and by making theoperational gestures correspond to actions a user's fingers cancomfortably perform. For instance, a user can slide a finger acrossslider 410 to adjust its position to a desired location. A user can tapa finger on the appropriate location to select a shadow option fromsection 406.

A different user interface can be deployed if the stylus is in theactive state. For example, FIG. 5 illustrates one example of a userinterface screen 500 that can be presented by a computing device (e.g.,computing device 100 of FIG. 1) according to an embodiment of thepresent invention. User interface screen 500, similarly to userinterface screen 300, supports creation and editing of a presentationthat can incorporate text, images, and other media. Screens 500 and 300can be implemented in the same application, with selection between thembeing based on whether a stylus in the active state is present.Similarly, to interface screen 300, a presentation page is rendered inmain section 502; in this example, the presentation page includes animage 504 and accompanying text 506. A control section 508 can providevarious user operable controls that the user can select (e.g., bytouching) to modify the presentation page. For example, when image 504is highlighted, selecting control button 510 can cause animage-formatting menu to be displayed; when text 506 is highlighted,selecting control 508 can cause a text-formatting menu to be displayed.

In some embodiments, user interface screen 500 can be presented when thestylus is in active mode, and controls in section 508 and/or menus thatare displayed can be adapted for stylus operation. For instance, if astylus is expected to have a smaller contact area than a finger, controlelements in section 508 can be smaller and/or more closely spaced thancontrols in section 308 of FIG. 3 without making it difficult for thestylus to unambiguously touch the desired control element. In addition,user interface screen 500 can present different control elements and/oror a different combination of control elements from screen 300. Forexample, in user interface screen 500, when a user selects image 504,control handles 512 can be displayed to facilitate precise resizing orother adjustments to image 504 using stylus 102.

In addition, control handles 512 can be used to invoke image propertiesmenus. By way of example, FIG. 6 shows an example of a user interfacescreen 600 that can be presented when a user, presented with userinterface screen 500 of FIG. 5, selects a control handle 512, e.g., bytapping with stylus 102. Interface screen 600 includes an imageformatting menu 602 that can appear as an overlay on top of a portion ofpresentation area 502. In this example, menu 602 provides variousoptions for display effects that can be associated with image 504. Forexample, the user can adjust the size, opacity, and shadow properties.In this example, a user can adjust each property using a correspondingvirtual rotational control (e.g., set-screw 604 to adjust photo size,set-screw 606 to adjust photo opacity, and so on).

A virtual set-screw (e.g., set-screw 604 or set-screw 606) can beoperated similarly to a real set-screw that is adjustable by insertingand twisting a screwdriver. In the case of a virtual set-screw, stylus102 can act as the “screwdriver.” While the use of a set-screw as acontrol paradigm may visually cue the user that the control element isoperable by rotation, other rotational control elements can bedisplayed, and in some instances, the option of rotational control neednot be visually indicated (examples of implied rotational controls aredescribed below).

Rotational control can be implemented in any user interface, providedthat rotational movement of stylus 102 about its longitudinal axis canbe detected. For example, as described above with reference to FIG. 2,an implementation of stylus 102 can include accelerometer(s) 236, whichcan be configured to detect rotational movement about the longitudinalaxis, and contact sensor(s) 232, which can be configured to detectcontact of the tip of stylus 102 with touchscreen 104. If rotationalmotion about the longitudinal axis is detected while the stylus tip isin contact with the touchscreen, stylus 102 can transmit a signalindicating rotational motion to computing device 100. The signal canindicate the amount and direction of rotation. Computing device 100 candetermine, e.g., based on the position of the contact location of stylus102 on touchscreen 104, which rotational control is being operated andinterpret the input accordingly, e.g., adjusting image size if set-screw604 is operated, adjusting image opacity if set-screw 606 is operated,and so on.

In some embodiments, the adjustment can be applied in real time inresponse to the rotational movement of the stylus. For example, therecan be a mapping between the change in the setting and the amount ofrotation detected. In some instances, the mapping can also take intoaccount the speed of rotation (e.g., adjusting a value by larger amountsfor a given amount of rotation if the rotation is made at high speed andby smaller amounts if the rotation is made at low speed).

In addition, value indicators 608, 610 (which indicate the current valuefor the corresponding setting) can be updated in real time. In someembodiments, a set-screw or other rotational control element can providea fine control over the values of various settings.

In some embodiments, when the stylus is in the active state and incontact with the surface, concurrent rotational and translationalmotions of the stylus can be detected and used to concurrently controlor adjust different properties of a displayed object. For example,referring to interface screen 500 of FIG. 5, a stylus can be used toadjust the position and rotation angle of image 504. FIG. 7 shows astylus 702 having a tip 704 in contact with image 504. Dashed line 706indicates a translational motion path for stylus tip 704, and dashedline 708 indicates a rotational motion path of stylus 702 about itslongitudinal axis. A user holding stylus 702 (the user's hand is notshown in order to avoid obscuring other elements) can concurrently orsuccessively execute paths 706 and 708.

FIG. 8 shows an example of a user interface screen 800 that can bepresented as a result of executing paths 706 and 708. Image 504 istranslated across the presentation page shown in main section 502 basedon path 706 such that the same point in image 504 remains in contactwith stylus tip 704. Image 504 is also rotated based on rotation path708. Thus, the user can adjust the position and rotation angle of adisplayed object. In this example, the option to rotate image 504 byrotating stylus 702 is not visually indicated in the user interface.Some embodiments can provide visual cues. For instance, when stylus tip704 contacts a point within image 504, a rotational knob or arrows orother indicator that the rotation angle of image 504 can be changedmight be presented.

As another example, a user can use simultaneous translation and rotationof a stylus to fine-tune a drawn object. FIG. 9 illustrates an exampleof a user interface screen 900 for a drawing program that can bepresented by a computing device (e.g., computing device 100 of FIG. 1)according to an embodiment of the present invention. Interface screen900 can provide a main drawing area 902, in which a user can drawobjects such as object 904. A control section 906 can provide varioususer-operable controls 908, e.g., to select a drawing tool (e.g.,different line properties, paintbrush properties, or the like), toperform functions such as filling an area with a color or pattern, tocopy and paste objects into a drawing, and so on.

In this example, object 904 is a curved object with a path defined usingBezier curves. Bezier curves are defined by a series of control points910, each of which has an associated tangent line (not shown in FIG. 9).

In certain embodiments of the present invention, translational motion ofa stylus can be used to adjust the position of a control point of aBezier curve while rotational motion can be used to adjust theorientation angle of the tangent line.

For instance, FIG. 10 shows an example of a user interface screen 1000that can be presented when tip 1002 of stylus 1004 is near to or incontact with a particular control point 1010. In this example, tangentline 1006 can appear in main drawing area 902. Translational motion oftip 1002 can adjust the position of control point 1010, while rotationalmotion of stylus 1004 about its longitudinal axis can adjust the angleof tangent line 1006. By way of illustration, dashed line 1012 indicatesa translational motion path for stylus tip 1002, and dashed line 1014indicates a rotational motion path of stylus 1004 about its longitudinalaxis. A user holding stylus 1004 (the user's hand is not shown in orderto avoid obscuring other elements) can concurrently or successivelyexecute paths 1012 and 1014.

FIG. 11 shows an example of a user interface screen 1100 that can bepresented as a result of executing paths 1012 and 1014. As a result ofexecuting translational path 1012, control point 1010 is moved to a newposition. As a result of executing rotational path 1014, the angle oftangent line 1006 is changed. Consequently, object 904′ has a somewhatdifferent shape from original object 904 of FIG. 9. The adjustment canbe made in real-time, and the user can translate and rotate stylus 1004to modify the shape of object 904 as desired. This provides an intuitiveinterface for fine-tuning curved lines in drawings.

As the examples of FIGS. 7-11 show, when a graphical user interface isdisplayed in the stylus mode, a computing device such as computingdevice 100 can concurrently detect translational motion of a contactpoint of a tip of the stylus with a surface of the touch-sensitivedisplay and rotational motion of the stylus about a longitudinal axis.In response, one property of a displayed object can be adjusted based onthe translational motion, while a second property of the displayedobject can be concurrently adjusted based on the rotational motion.

It will be appreciated that the user interface screens of FIGS. 3-11 areillustrative and that variations and modifications are possible. A userinterface screen can include any number and arrangement of inputelements. Input elements can be used for a variety of purposes,including editing displayed objects, defining properties of displayedobjects (shadows, opacity, sizes, fonts, colors, orientations, etc.),defining objects to be displayed (e.g., entering text, selecting imagesor other content, drawing) specifying operations to perform (e.g., mediaplayback operation such as play/pause, previous track), adjustingproperties of output devices (e.g., volume control, brightness control),retrieving and presenting content (e.g., based on a user selectinghyperlinks in a web browser), invoking functionality (e.g., launching anapplication), and any other purpose for which a user may interact with acomputing device having a touchscreen display.

The same application can present different user interface screens, e.g.,depending on whether the stylus is in the active state or the idlestate. In some instances, an application can dynamically switch to adifferent interface screen if the state of the stylus changes. Forinstance, a presentation editing application can present interfacescreen 300 for as long as the stylus is in an idle state and switch tointerface screen 500 if the stylus enters the active state.

FIG. 12 is a flow diagram of a process 1200 for rendering a userinterface according to an embodiment of the present invention. Process1200 can be implemented, e.g., in an application executing on computingdevice 100 of FIG. 1.

At block 1202, process 1200 can obtain stylus data. For example, asdescribed above, stylus 102 can send a signal to computing device 100,and process 1200 can receive this signal. At block 1204, process 1200can determine the state of the stylus (e.g., active or idle). Forexample, as described above, the signal sent by stylus 102 can include amessage indicating the stylus state, which stylus 102 can determinebased on sensor readings. As another example, stylus 102 can send asignal that includes the sensor data, and process 1200 or anotherprocess on computing device 100 (e.g., an operating system process) candetermine the state of the stylus based on the received signal.

At block 1206, a determination is made as to whether stylus 102 is inthe active state. If so, then at block 1208, the user interface can berendered in a “stylus mode.” For example, the code for rendering theuser interface can include conditional statements that can cause certaininput elements to be rendered in a different manner (e.g., slider versusset-screw) or to be rendered or not rendered (e.g., control handles 512of FIG. 5) depending on whether the stylus is in the active state.

If, at block 1206, stylus 102 is not in the active state (e.g., ifstylus 102 is in the idle state), then at block 1210, the user interfacecan be rendered in a “finger mode.” For example, as noted above, thecode for rendering the user interface can include conditional statementsgoverning how input elements are rendered, and block 1210 can includeexecuting the same code as block 1208, with the difference in conditionresulting in a different rendering.

The difference between user interfaces rendered at blocks 1208 and 1210can include a difference in the rendering of at least one user-operableinput element of the user interface. For instance, the stylus mode(block 1208) can provide a rendering of the element that is based on anexpectation that a stylus will be used to operate the element, while thefinger mode (block 1210) can provide a rendering of the element that isbased on an expectation that a finger will be used to operate theelement. Thus, for example, an element can be rendered at a smaller sizein the stylus mode than in the finger mode. An element can be renderedfor a different type of operation in the stylus mode as opposed to thefinger mode (e.g., an adjustable control can be rendered as a set-screwin the stylus mode and as a slider in the finger mode). The renderingscan also differ in the number and placement of user-operable inputelements.

It is to be understood that rendering a user interface or user interfaceelement in a mode associated with a particular input tool (e.g., astylus mode associated with a stylus) need not preclude the use of otherinput tools (e.g., a finger) to interact with the interface or element.Accordingly, a user might hold the stylus in one hand and touch thescreen with fingers of the other hand, or the user might hold the stylusin a manner that leaves one or more fingers available to touch thescreen, or one user might be holding the stylus while another usertouches the screen with fingers. In such instances, the graphical userinterface can remain in stylus mode for as long as the user continues tohold the stylus. Alternatively, depending on how the touch sensors ofthe stylus are arranged, it can be possible to determine whether thecontact with the user's hand corresponds to holding the stylus in anorientation associated with actual use (e.g., a contact patternassociated with holding a pencil or pen) or a non-use orientation (e.g.,wedged between two fingers), and the mode to be used for rendering theinterface can be selected accordingly.

In some instances, a stylus might not be present. For instance, inembodiments described above, a computing device can detect the presenceof a stylus based on transmissions from the stylus; if no transmissionsare received, then the stylus can be considered to be absent. This casecan be handled, e.g., by assuming the idle state for the stylus in theabsence of transmissions. As another example, in some embodiments, thestylus and the computing device can establish a pairing (e.g., aBluetooth pairing), and the computing device can ignore anystatus-related messages from any stylus with which it is not paired.Thus, a user's experience need not be affected by the mere presence of astylus in the vicinity; a particular device would enter the stylus modeonly if it is paired with a stylus that is in the active state.

It will be appreciated that process 1200 is illustrative and thatvariations and modifications are possible. Steps described as sequentialmay be executed in parallel, order of steps may be varied, and steps maybe modified, combined, added or omitted. For instance, the determinationof the state of the stylus can be made by the computing device or thestylus.

FIG. 13 is a flow diagram of a process 1300 for determining stylus stateaccording to an embodiment of the present invention. Process 1300 can beimplemented, e.g., in stylus 102 of FIG. 1. As described above withreference to FIG. 2, an implementation of stylus 102 can include controllogic to interpret signals from touch sensors (e.g., capacitive and/orpressure-based touch sensors) and/or motion sensors to determine whetherit is being held by a user.

At block 1302, the control logic of stylus 102 can receive sensor data(e.g., from touch sensors 114 and/or internal motion sensors). At block1304, the sensor data can be analyzed to determine whether stylus 102 isin a user's hand. For example, touch sensor data can be analyzed todetect a pattern of contact areas consistent with a human hand. In someembodiments, the analysis can include comparing the pattern of contactareas to patterns associated with a human hand in a particular pose(e.g., holding a pencil or pen in order to write). As another example,motion sensor data can be used to detect movement consistent with beingheld by a person as opposed to being at rest on a surface. Suchdistinctions can be based on amplitude and/or frequency analysis of theaccelerometer data. In some embodiments, a combination of touch sensorand motion sensor data can be used.

At block 1306, if it is determined that stylus 102 is in a user's hand,then at block 1308, stylus 102 can transmit a signal to computing device100 indicating that it is in the active state. If, at block 1306, stylus102 is not determined to be in a user's hand, then at block 1310, stylus102 can transmit a signal to computing device 100 indicating that it isin the idle state.

Process 1300 can be performed iteratively, e.g., at regular intervalssuch as every 0.1 seconds or every 0.01 seconds, or every 0.001 seconds,and stylus 102 can periodically report its state. In some embodiments,transmission of a signal at block 1308 or 1310 occurs only if the statehas changed since the last execution of process 1300. Where transmissionoccurs on a state change, computing device 100 can assume that the stateis unchanged until the next transmission. In some embodiments, computingdevice 100 can send a message to stylus 102 requesting the currentstate, and process 1300 can be executed in response to such a request.For example, computing device 100 might send a state-requesting messageif it has not received a state message from stylus 102 within some“heartbeat” period, or computing device 100 might send astate-requesting message if it is preparing to render a new userinterface or otherwise update the displayed image.

It is to be understood that process 1300 can be but need not beperformed by the stylus. In some embodiments, stylus 102 can transmitits sensor data to computing device 100 without analysis, and computingdevice 100 can perform the analysis and determine whether the stylus isin the active or idle state.

As noted above, stylus state can be determined based on sensors thatrespond to touching of particular surfaces of the stylus (e.g., a barrelportion in the case of an elongate stylus as shown in FIG. 1) and/ormotion sensors such as accelerometers. Any type and combination ofsensor data can be used. For instance, accelerometers or other motionsensors can be used in the absence of touch sensors, touch sensors canbe used in the absence of motion sensors, or a combination of motionsensor data and touch sensor data can be used. In some instances, whereboth accelerometer data and touch sensor data are used, a signalindicating that the stylus is being held by a user from either sensorsuffices to trigger the active state; this can be useful, e.g., if auser holds the stylus using a prosthetic device that the touch sensordoes not reliably register.

While the invention has been described with respect to specificembodiments, one skilled in the art will recognize that numerousmodifications are possible. The computing device and the stylus can varyas to form factor, size and components. Any computing device having atouchscreen display can be used. A stylus can include any tool that canoperate a touchscreen display of a computing device by direct contact ornear-contact and that is capable of communicating sensor data and/orstate information to the computing device. In the examples shown above,a stylus communicates wirelessly with the computing device, e.g., usingBluetooth or Bluetooth LE or the like. However, a wired connection canbe implemented. For example, a stylus can be connected into a cable thatplugs into a receptacle connector in the housing of a computing device;the cable can be sufficiently long and flexible to avoid interferingwith use of the computing device. Where a wired connection is used, thestylus can draw power from the computing device or from its own internalpower source. In the case of a wireless connection, the stylus can haveits own internal power source; in some instances, it may be possible tophysically connect the stylus to the computing device for recharging ofthe stylus, and in some instances, the stylus may support both wired andwireless communication interfaces.

In some embodiments, the stylus can include one or more user-operableinput controls such as buttons, switches, dials or the like, andcommunication from the stylus to the computing device can includemessages indicating the status and/or operation of any input controlsthat may be present on the stylus.

In embodiments described above, a computing device can use the active oridle state of a stylus to affect rendering of a user interface. Thisstate information can also be used for other purposes. For example, ifthe stylus is in the idle state, the stylus can also reduce its powerconsumption, e.g., by powering down some of its sensors anddiscontinuing or reducing transmissions to the computing device. Atleast one sensor can remain operational in order to detect a possibletransition to the active state, and this event can result in waking upother sensors that can confirm the transition. As another example, ifthe user picks up the stylus while the computing device is in alow-power state (e.g., with its touchscreen display turned off), thestylus can notify the computing device, and the computing device cantransition to a ready-to-use state, e.g., turning on its touchscreendisplay.

Embodiments of the present invention can be realized using anycombination of dedicated components and/or programmable processorsand/or other programmable devices. The various processes describedherein can be implemented on the same processor or different processorsin any combination. Where components are described as being configuredto perform certain operations, such configuration can be accomplished,e.g., by designing electronic circuits to perform the operation, byprogramming programmable electronic circuits (such as microprocessors)to perform the operation, or any combination thereof. Further, while theembodiments described above may make reference to specific hardware andsoftware components, those skilled in the art will appreciate thatdifferent combinations of hardware and/or software components may alsobe used and that particular operations described as being implemented inhardware might also be implemented in software or vice versa.

Computer programs incorporating various features of the presentinvention may be encoded and stored on various computer readable storagemedia; suitable media include magnetic disk or tape, optical storagemedia such as compact disk (CD) or DVD (digital versatile disk), flashmemory, and other non-transitory media. (It is understood that “storage”of data is distinct from propagation of data using transitory media suchas carrier waves.) Computer readable storage media encoded with theprogram code may be packaged with a compatible electronic device, or theprogram code may be provided separately from electronic devices (e.g.,via Internet download or as a separately packaged computer-readablestorage medium).

Thus, although the invention has been described with respect to specificembodiments, it will be appreciated that the invention is intended tocover all modifications and equivalents within the scope of thefollowing claims.

What is claimed is:
 1. A method, comprising: at an electronic device that includes a display device and is in communication with a stylus that includes touch sensors: while the stylus is in a hand of a user: while the stylus is being held in a first manner in the hand of the user, displaying a user interface via the display device; receiving one or more wireless communications from the stylus that indicate a change in state of the stylus to being held in a second manner that is different from being held in the first manner; and in response to receiving the one or more wireless communication from the stylus, updating the displayed user interface in accordance with the change in state of stylus from being held in the first manner to being held in the second manner.
 2. The method of claim 1, wherein the stylus is an idle state when the stylus is being held in the first manner and, while in the idle state, is not used to provide input to the electronic device; and the stylus is an active state when the stylus is being held in the second manner, and while in the active state, the stylus is used to provide input to the electronic device.
 3. The method of claim 2, including determining a change in state of the stylus from the idle state to the active state in accordance with a determination that a contact pattern of the user's hand with the touch sensors of the stylus corresponds to holding the stylus in an orientation associated with use of the stylus.
 4. The method of claim 3, including determining that the stylus is in the idle state in accordance with a determination that a contact pattern of the user's hand with the touch sensors of the stylus corresponds to holding the stylus in an orientation associated with non-use of the stylus.
 5. The method of claim 1, including, while the stylus is being held in the first manner in the hand of the user, displaying the user interface with a first appearance; and while the stylus is being held in the second manner in the hand of the user, displaying the user interface with a second appearance different from the first appearance.
 6. The method of claim 5, wherein the first appearance includes one or more input tools associated with using a finger for input, and the second appearance includes one or more input tools associated with use of the stylus for input.
 7. The method of claim 6, wherein the user interface with the first appearance and the user interface with the second appearance are for a same application.
 8. The method of claim 5, wherein the first appearance includes a first type of control for adjusting a first property of a user interface object, and the second appearance includes a second type of control for adjusting the first property of the user interface object, and the second type of control is different from the first type of control.
 9. An electronic device, comprising: one or more processors; a display device; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: while a stylus, which is in communication with the electronic device, is in a hand of a user: while the stylus is being held in a first manner in the hand of the user, displaying a user interface via the display device; receiving one or more wireless communications from the stylus that indicate a change in state of the stylus to being held in a second manner that is different from being held in the first manner; and in response to receiving the one or more wireless communication from the stylus, updating the displayed user interface in accordance with the change in state of stylus from being held in the first manner to being held in the second manner.
 10. The electronic device of claim 9, wherein the stylus is an idle state when the stylus is being held in the first manner and, while in the idle state, is not used to provide input to the electronic device; and the stylus is an active state when the stylus is being held in the second manner, and while in the active state, the stylus is used to provide input to the electronic device.
 11. The electronic device of claim 10, wherein the stylus includes touch sensors, and the one or more programs include instructions for determining a change in state of the stylus from the idle state to the active state in accordance with a determination that a contact pattern of the user's hand with the touch sensors of the stylus corresponds to holding the stylus in an orientation associated with use of the stylus.
 12. The electronic device of claim 11, wherein the one or more programs include instructions for determining that the stylus is in the idle state in accordance with a determination that a contact pattern of the user's hand with the touch sensors of the stylus corresponds to holding the stylus in an orientation associated with non-use of the stylus.
 13. The electronic device of claim 9, wherein the one or more programs include instructions for: while the stylus is being held in the first manner in the hand of the user, displaying the user interface with a first appearance; and while the stylus is being held in the second manner in the hand of the user, displaying the user interface with a second appearance different from the first appearance.
 14. The electronic device of claim 13, wherein the first appearance includes one or more input tools associated with using a finger for input, and the second appearance includes one or more input tools associated with use of the stylus for input.
 15. The electronic device of claim 14, wherein the user interface with the first appearance and the user interface with the second appearance are for a same application.
 16. The electronic device of claim 13, wherein the first appearance includes a first type of control for adjusting a first property of a user interface object, and the second appearance includes a second type of control for adjusting the first property of the user interface object, and the second type of control is different from the first type of control.
 17. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device that includes a display device and that is in communication with a stylus having touch sensors, cause the electronic device to: while the stylus is in a hand of a user: while the stylus is being held in a first manner in the hand of the user, display a user interface via the display device; receive one or more wireless communications from the stylus that indicate a change in state of the stylus to being held in a second manner that is different from being held in the first manner; and in response to receiving the one or more wireless communication from the stylus, update the displayed user interface in accordance with the change in state of stylus from being held in the first manner to being held in the second manner.
 18. The non-transitory computer readable storage medium of claim 17, wherein the stylus is an idle state when the stylus is being held in the first manner and, while in the idle state, is not used to provide input to the electronic device; and the stylus is an active state when the stylus is being held in the second manner, and while in the active state, the stylus is used to provide input to the electronic device.
 19. The non-transitory computer readable storage medium of claim 18, wherein the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to determine a change in state of the stylus from the idle state to the active state in accordance with a determination that a contact pattern of the user's hand with the touch sensors of the stylus corresponds to holding the stylus in an orientation associated with use of the stylus.
 20. The non-transitory computer readable storage medium of claim 19, wherein the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to determine that the stylus is in the idle state in accordance with a determination that a contact pattern of the user's hand with the touch sensors of the stylus corresponds to holding the stylus in an orientation associated with non-use of the stylus.
 21. The non-transitory computer readable storage medium of claim 17, wherein the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to: while the stylus is being held in the first manner in the hand of the user, display the user interface with a first appearance; and while the stylus is being held in the second manner in the hand of the user, display the user interface with a second appearance different from the first appearance.
 22. The non-transitory computer readable storage medium of claim 21, wherein the first appearance includes one or more input tools associated with using a finger for input, and the second appearance includes one or more input tools associated with use of the stylus for input.
 23. The non-transitory computer readable storage medium of claim 22, wherein the user interface with the first appearance and the user interface with the second appearance are for a same application.
 24. The non-transitory computer readable storage medium of claim 21, wherein the first appearance includes a first type of control for adjusting a first property of a user interface object, and the second appearance includes a second type of control for adjusting the first property of the user interface object, and the second type of control is different from the first type of control. 